REMARKS 

Applicant respectfully requests reconsideration and allowance of the 
subject application. Claims 8, 32, 35 and 55 are amended. New claims 57-66 are 
added. Claims 1-30, 32-38, 40-44 and 46-66 are pending in this application. 

The amendment to the specification addresses minor informalities noted 
during review. No new matter is added by the amendment to the specification. 

35 ILS.C.S112 

Claims 32-38, 55 and 56 stand rejected under 35 U.S.C. §112, second 
paragraph. Claims 32 and 55 have been amended to obviate the concerns noted in 
the Office Action, however, these amendments are not intended to alter the scope 
of the claims. Accordingly, Applicant respectfully requests that the §112 
rejections be withdrawn. 
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Art Rejections 

Claims 1, 8, 9, 32, 35-38, 40-44 and 48-51 stand rejected under 35 U.S.C. 
102(b) as being anticipated by "Relevance Feedback Techniques in Interactive 
Content-Based Image Retrieval" by Rui et al. (Jour. Vis. Comm. and Image Rep., 
Vol. 10, pp. 39-62, March 1999) (hereinafter "RFT"). Claims 11-18, 46 and 47 
stand rejected under 35 U.S.C. 102(a) as being anticipated by "MindReader: 
Querying Databases Through Multiple Examples" by Ishikawa et al. (Proc. 24 
VDLB Conf. (New York), 1988, pp. 218-227) (hereinafter "MR"). 

Claims 10 and 52 stand rejected under 35 U.S.C. § 103(a) as being 
unpatentable over RFT. Claims 2-7, 19-30, 33, 34 and 53-56 stand rejected under 
35 U.S.C. § 103(a) as being unpatentable over RFT in view of MR. Applicant 
respectfully traverses the anticipation and unpatentability rejections and requests, 
reconsideration. 
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35 U.S.C. fi 102 

Anticipation is a legal term of art. Applicant notes that in order to provide 
a valid finding of anticipation, several conditions must be met: (i) the reference 
must include every element of the claim within the four corners of the reference 
(see MPEP §2121); (ii) the elements must be set forth as they are recited in the 
claim (see MPEP §2131); (iii) the teachings of the reference cannot be modified 
(see MPEP §706.02, stating that "No question of obviousness is present" in 
conjunction with anticipation); and (iv) the reference must enable the invention as 
recited in the claim (see MPEP §2121.01). Additionally, (v) these conditions must 
be simultaneously satisfied. 

The §102 rejection of claims 1, 8, 9, 11-18, 32, 35-38, 40-44 and 46-51 is 
believed to be in error. Specifically, the PTO and Federal Circuit provide that 
§102 anticipation requires that each and every element of the claimed invention be 
disclosed in a single prior art reference . In re Spada, 911 F.2d 705, 15 USPQ2d 
1655 (Fed. Cir. 1990). The corollary of this rule is that the absence from a cited 
§102 reference of any claimed element negates the anticipation. Kloster 
Speedsteel AB, et al v. Crucible, Inc., et al. 9 793 F.2d 1565, 230 USPQ 81 (Fed. 
Cir. 1986). 

No §103 rejection has been lodged regarding claims 1, 8, 9, 11-18, 32, 35- 
38, 40-44 and 46-51. Accordingly, if Applicant can demonstrate that RFT does 
not disclose any one claimed element with respect to claims 1, 8, 9, 32, 33, 36 and 
37, or that MR does not disclose any one claimed element with respect to claims 
11-18, 46 and 47, the associated §102 rejection must be withdrawn, and a 
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subsequent non-Final action made with a different rejection in the event that the 
Examiner still finds such claim(s) to be not allowable. 

RFT describes relevance feedback techniques in interactive content-based 
image retrieval (Title). RFT describes an algorithm (Calcs. 3-8) wherein a query 
object Q is described in terms of features F having associated presentation vectors 
rjj (Calcs. 3-5 and associated text). The user's information need is distributed 
among different features fj of the query object Q, according to their corresponding 
weights Wy. The various objects' similarities to the query Q is calculated 
according to a similarity measure my and the weights Wjj k (Calc. 6) and feature 
similarity values are calculated (Calc. 7). An overall similarity is then determined 
(Calc. 8), and the objects in the database are ordered by their overall similarity to 
Q (item 7) and then are ranked by the user as to relevancy (item 8). The system 
updates the weights according to the user's feedback such that the adjusted Q is a 
better approximation to the user's information need (item 9), and the system 
iterates the comparison of the adjusted Q to the objects in the database (item 10). 

MR describes a method for querying databases through multiple examples 
(Title). MR teaches that the depends on incorporation of user feedback to 
calculate a symmetric distance matrix M (sec. 3.2), which represents a generalized 
ellipsoid distance function (Table 1, Figure 2). When an inverse to a weighed 
covariance matrix C exists, the matrix M that minimizes Calc. 4 is: M = 
(det(C)) l/n C _1 (Theorem 2, page 221, Appendix B). When M is restricted to 
diagonal matrices only, M has elements mjj a l/(oj ). When the covariance matrix 
C is singular and non-invertible (i.e., when C" 1 does not exist), a pseudo-inverse 
matrix or Moore-Penrose matrix is used (Theorem 3, Appendix D). 
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In contrast to RFT, claim 1 recites "One or more computer readable media 
having stored thereon a plurality of instructions that, when executed by one or 
more processors, causes the one or more processors to perform acts including: 
receiving an initial image selection; generating a plurality of query vectors by 
extracting , for each query vector, one of a plurality of low-level features from the 
initial image selection ; selecting a set of potentially relevant images based at least 
in part on distances between the plurality of query vectors and a plurality of 
feature vectors corresponding to low-level features of a plurality of images; 
receiving feedback regarding the relevance of one or more images of the set of 
potentially relevant images; generating a new plurality of query vectors based at 
least in part on the feedback : generating a weighting of feature elements based at 
least in part on the feedback; and selecting a new set of potentially relevant images- 
based at least in part on both the weighting of feature .elements and distances 
between the new plurality of query vectors and the plurality of feature vectors", 
which is not taught or disclosed by RFT. 

RFT does not teach or disclose any computer readable media having stored 
thereon a plurality of instructions that, when executed by one or more processors, 
cause the one or more processors to perform acts as recited in claim 1. RFT is 
silent with respect to implementation of any of the ideas described by RFT. 

In other words, RFT does not provide any description regarding 
implementation whatsoever. Examples of approaches that might be employed 
could include an ASIC or neural network or an analog computer, none of which 
involve computer readable media or a computer program that is executable by a 
processor. 
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RFT does not teach or disclose any computer readable media including a 
computer program that is executable by a processor. Put another way, MR is 
completely void of the language "computer readable media" or "computer 
program", or, for that matter, "processor". As such, RFT does not provide the 
elements recited in claim 1 and cannot possibly enable the recitation of claim 1 . 

Further, RFT teaches (p. 3, section 2) that the object O is represented as O 
= 0(D, F, R). D, F and R correspond to predetermined quantities, respectively 
identified by RFT (p. 3, section 2, "The Multimedia Object Model") as follows: 

• D is the raw image data, e.g., a JPEG image. 

• F = {fi} is a set of low-level visual features associated with 
the image object, such as color, texture and shape. 

• R = {rjj} is a set of representations for a given feature fi .... 
RFT explicitly states (p. 3, last sentence) that "a query Q has the same model as 
that of the image objects". 

RFT explicitly teaches (p. 4, section 3) that "An image object model O, 
together with a set of similarity measures M = {my}, specifies a CBIR model (D, 
F, R, M). The similarity measures are used to determine how similar or dissimilar 
two objects are." RFT also teaches (p. 4, step 2) that "The user's information need, 
represented by the query object Q, is distributed among different features f u 
according to their corresponding weights Wy" RFT teaches (section 3.1) updating 
the weights to reflect different user emphases, stating (p. 4, step 9) that "The 
system updates the weights according to the user's feedback such that the adjusted 
Q is a better approximation to the user's information need." 
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In other words, the portion of the query Q that is modified is other than the 
information associated with the feature vectors £ of the features F. As such, RFT 
does not teach "generating a plurality of query vectors by extracting, for each 
query vector, one of a plurality of low-level features from the initial image 
selection" after receiving the query vectors, as recited in claim 1. RFT teaches that 
the query objects include the set of low-level features F that are characterized by 
the feature vectors f i? as do the data objects. RFT does not teach or disclose 
modification of the properties D, F or R in response to initial image selection or 
subsequent thereto. 

As such, RFT does not teach "generating a plurality of query vectors by 
extracting, for each query vector, one of a plurality of low-level features from the 
initial image selection", as recited in claim 1 ; 

RFT further teaches that the user feedback is solely reflected in the weights, 
stating (p. 3) that "The weights are essential in modeling high level concepts and 
perception subjectivity. Section 3 discusses how the weights are dynamically 
updated based on the relevance feedback to track user's information need." As 
such, RFT does not and cannot teach "generating a new plurality of query vectors 
based at least in part on the feedback", as recited in claim 1. 

In contrast to RFT, claim 8 , as amended, recites "One or more computer 
readable media having stored thereon a plurality of instructions that, when 
executed by one or more processors, causes the one or more processors to perform 
acts including: receiving an initial image selection; generating a plurality of query 
vectors by extracting, for each query vector, one of a plurality of low-level 
features from the initial image selection; selecting a set of potentially relevant 
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images based at least in part on distances between the plurality of query vectors 
and a plurality of feature vectors corresponding to low-level features of a plurality 
of images; receiving feedback regarding the relevance of one or more images of 
the set of potentially relevant images; generating a new plurality of query vectors 
based at least in part on the feedback; generating a weighting of feature elements 
based at least in part on the feedback; and selecting a new set of potentially 
relevant images based at least in part on both the weighting of feature elements 
and distances between the new plurality of query vectors and the plurality of 
feature vectors, wherein f. represents a summation, over the images in the set of 
potentially relevant images, of a product of a relevance of the image and a distance 
between the query vector and the feature vector, and wherein the selecting a new 
set of potentially relevant images comprises combining, for each image, a 
weighted distance between the plurality of query vectors and the plurality of. 
feature vectors, and wherein the weight ( u i ) for each of a plurality ( / ) of distances 
between a query vector and a corresponding feature vector is calculated as: 



which is not taught or disclosed by RFT. 

RFT does not teach or disclose "One or more computer readable media 
having stored thereon a plurality of instructions that, when executed by one or 
more processors, causes the one or more processors to", as recited in claim 8. 
RFT does not teach or disclose "generating a plurality of query vectors by 
extracting, for each query vector, one of a plurality of low-level features from the 
initial image selection", as recited in claim 8. RFT also does not teach or disclose 
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"generating a new plurality of query vectors based at least in part on the 
feedback", as recited in claim 8. 

In contrast to RFT, claim 48 recites "A method of generating a weight to 
apply to distances between query vectors and feature vectors when combining the 
distances, the method comprising: receiving feedback regarding the relevance of 
each image of a set of images; wherein f. represents a summation, over the 

images in the set of images, of a product of a relevance of the image and a distance 
between the query vector and the feature vector; and generating a weight (w.) for 

each of a plurality ( / ) of distances between a query vector corresponding to one 
of a plurality ( / ) of features and a feature vector corresponding to the one of the 
plurality ( / ) of features as: 



which is not taught or disclosed by RFT. 

The weights w, calculated as recited in claims 8 and 48 are a summation of 
square roots of summations of products of relevance and distance between the 
query vector and the feature vector. A square root operation is not a linear 
operation. 

RFT explicitly discloses that the weights Wy are formed using linear 
techniques (Calcs. 17-19), are normalized using linear operations (Calc. 20) and 
are modified linearly (paragraph immediately preceding Calc. 9). RFT teaches 
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that use of such linear techniques allows two levels of calculation to be combined 
into one, as represented in Calc. 9. 

Claim 32 recites "A method comprising: for one of a plurality of images 
and each of a plurality of features , generating, based on a set of search criteria, a 
query vector for the feature, identifying a feature vector, corresponding to the 
image, for the feature, and determining how closely the feature vector matches the 
query vector ; and determining how closely the image matches the set of search 
criteria based on how closely, for the plurality of features, the feature vectors 
match the query vectors , wherein generating the query vector comprises 
generating the query vector based at least in part on user relevance feedback 
regarding how relevant images previously displayed to a user were", which is not 
taught or disclosed by RFT. 

RFT describes an approach whereby an object's similarity to a query object 
is evaluated (step 4), the representation's similarity values are combined into a 
feature's similarity value (weight Wy for the feature, step 5) and these similarities 
are then combined (step 6) to provide an overall similarity. This overall similarity 
is then employed to evaluate rank ordering (step 7, stating that "The objects in the 
database are ordered by their overall similarity to Q."). 

In contrast, the process recited in claim 32 reflects a hierarchical approach 
to determining how closely two images match. First individual distances between 
the feature vectors and the query vectors are determined, and then these individual 
distances are combined. 

Further, RFT does not teach or disclose " generating the query vector based 
at least in part on user relevance feedback " together with " determining how 
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closely the feature vector matches the query vector ", as recited in claim 32, 
because RFT teaches modification of weights for query feature vectors. The 
modified weights are used in comparing predetermined query feature vectors to 
feature vectors of the objects. 

Claim 41 recites "One or more computer readable media having stored 
thereon a plurality of instructions that, when executed by one or more processors, 
causes the one or more processors to perform acts including: identifying a plurality 
of query vectors for one image, each query vector corresponding to one of a 
plurality of features; identifying a plurality of feature vectors for another image, 
each feature vector corresponding to one of the plurality of features; for each 
feature, determining a distance between the corresponding query vector and the 
corresponding feature vector ; and combining the distances to generate a value 
representing an overall distance between the one and the another image, wherein 
the identifying the plurality of query vectors comprises generating the plurality of 
query vectors based at least in part on user relevance feedback regarding how 
relevant images previously displayed to a user were", which is not taught or 
disclosed by RFT. As noted above, determining a distance, of a feature-by-feature 
basis, as recited in claim 41, reflects a hierarchical approach, while RFT 
determines overall similarity. 

Further, RFT does not teach or disclose " generating the query vector based 
at least in part on user relevance feedback " together with " determining how 
closely the feature vector matches the query vector ", as recited in claim 32, 
because RFT teaches modification of weights for query feature vectors. The 
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modified weights are used in comparing predetermined query feature vectors to 
feature vectors of the objects. 

Claim 50 recites "A system comprising: a client device; a collection of a 
plurality of images; an image server, coupled to the client device and the 
collection of a plurality of images, the image server to receive image retrieval 
requests from the client device and to, receive an initial image selection from the 
client device, generate a plurality of query vectors by extracting, for each query 
vector, one of a plurality of low-level features from the initial image selection, 
select a set of potentially relevant images based at least in part on distances 
between the plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images, receive feedback 
regarding the relevance of one or more images of the set of potentially relevant 
images, generate a new plurality of query vectors based at least in part on the 
feedback, generate a weighting of feature elements based at least in part on the 
feedback, and select a new set of potentially relevant images based at least in part 
on both the weighting of feature elements and distances between the new plurality 
of query vectors and the plurality of feature vectors", which is not taught or 
disclosed by RFT. 

As noted above with respect to claim 1, RFT does not teach or disclose 
"image server to ... generate a plurality of query vectors by extracting , for each 
query vector, one of a plurality of low-level features from the initial image 
selection ", as recited in claim 50. RFT instead teaches computing distances from 
image properties of the image model 0(D, F, R). There is no teaching in RFT of 
extracting low-level features F. As also noted above with respect to claim 1, 
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RFT does not teach or disclose an image server to "generate a new plurality of 
query vectors based at least in part on the feedback", as recited in claim 50. RFT 
instead teaches modification of weights only. RFT further does not teach or 
disclose anything to "select a new set of potentially relevant images based at least 
in part on both the weighting of feature elements and distances between the new 
plurality of query vectors and the plurality of feature vectors ", as recited in claim 
50. RFT teaches modification of the weights to incorporate user relevancy 
feedback. 

Claim 51 recites "One or more computer readable media as recited in claim 
1, wherein the receiving feedback comprises receiving feedback in a range 
including at least Highly Relevant, Relevant, No Opinion, Irrelevant, and Highly 
Irrelevant", which is not taught or disclosed by RFT. As noted above with respect 
to claim 1 , RFT is void of any teaching or disclosure regarding implementation of 
the ideas presented in RFT. 

Claim 11 recites "A method of selecting between two types of matrixes to 
be used to weight, based on relevance feedback, a plurality of feature elements for 
image retrieval, the method comprising: selecting one of the two types of matrixes 
based on both a number of previously retrieved relevant images and a length of a 
feature vector including the plurality of feature elements", which is not taught or 
disclosed by MR. 

The Office Action states (p. 8) that MR discloses a method of selecting 
between two types of matrixes as claimed. See section 3 on pages 220-221 for this 
disclosure. Refer specifically to sections 3.2-3.4 and Appendix D for the details 
of this disclosure. In particular, MR teaches "a method of selection [See section 
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3.4] between two types of matrix (a full covariance matrix OR Moore-Penrose 
inverse matrix] to be used to weight, based on relevance feedback, a plurality of 
feature elements for image retrieval Applicant disagrees and requests 

reconsideration. 

MR discloses (section 3) use of a symmetric matrix M as a generalized 
ellipsoid distance matrix relating to distances between a query point described by a 
vector q and sample vectors x in a data point matrix X. MR provides no teaching 
or disclosure of "a plurality of feature elements for image retrieval" as recited in 
claim 1 1 . MR also provides no disclosure of use of "selecting between two types 
of matrixes to be used to weight" such, as recited in claim 1 1 . 

MR discloses several examples: finding information relating to "mildly 
overweight" people (p. 219, left column, Fig. 1) based on weight divided by height 
in a data system characterized by weight and height coordinates (a "diagonal 
inquiry") and finding points near a road segment using the Mongtomery County 
dataset, comprising endpoints of road segments in Montgomery County, Maryland 
(p. 223, left hand column; Fig. 6). These are both two-dimensional datasets; 
neither involves images. Other areas where MR indicates the model will be useful 
include multimedia systems and digital libraries, general approximate matching in 
traditional databases, time sequences ("find stocks similar to, e.g., IBM's stock") 
and spatial databases ("find gas stations close to the 1-270 interstate highway"). 
MR provides no indication of what kinds of search criteria or data would be useful 
in these contexts - keywords, human-defined numerical rankings. 

Further, MR does not teach or disclose choosing between two types of 
matrixes, as recited in claim 11. MR teaches (sections 3.3 and 3.4) selection 
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between three types of matrixes: a full covariance matrix (theorems 1 and 2, 
appendix B), a diagonal matrix, or a pseudo-inverse or Moore-Penrose matrix 
(section 3.4, appendix C). 

Claim 12 recites that "the selecting comprises selecting one of the two types 
of matrixes based on both a number of previously retrieved potentially relevant 
images which were identified by a user as being relevant, and the length of the 
feature vector including the plurality of feature elements", which is not taught or 
disclosed by MR. The Office Action cites Table I and section 3.2 with respect to 
claim 12. Table I is void of any reference whatsoever to any number of previously 
retrieved images or of potentially relevant images. Sections 3.2-3.4 do not cure 
these deficiencies. 

Claim 13 recites that "the plurality of feature elements are all elements of ; 
the same feature". The sample vectors x described by MR may be, for example, 
height and weight data for an individual - different features of the same subject or 
element , and not "elements of the same feature" as recited in claim 13. 

In contrast to MR, claim 14 recites that "the selecting comprises using a 
first type of matrix if the number of retrieved relevant images is less than the 
length of the feature vector, and otherwise using a second type of matrix", while 
claim 15 recites that " the first type of matrix comprises a diagonal matrix and 
wherein the second type of matrix comprises a full matrix ", which aspects are not 
taught or disclosed by MR. 

MR discloses that when the covariance matrix inverse C" 1 exists, either a 
matrix M = (det(C)) 1/n C" ! minimizes Calc. 4 or, if the matrix M is restricted to 
being diagonal, Calc. 9 (M has elements my a l/(aj 2 )) minimizes Calc. 4 (theorem 



lee®hayes p* »9-32*-92S6 



42 



Application No. 09/660,536 



2). MR discloses that when the number of feedback points is less than the number 
of feature dimensions, the Moore-Penrose inverse matrix satisfies Calc. 4. MR 
teaches use of either a full or diagonal matrix OR the use of the Moore-Penrose 
matrix and does not provide any selection criteria for when to discriminate 
between the first two options. 

In contrast to MR, claim 16 recites that "the selecting comprises using a 
first type of matrix if the length of the feature vector exceeds the number of 
retrieved relevant images by at least a threshold amount , and otherwise using a 
second type of matrix", which is not taught or disclosed by MR. 

The Office Action states that MR discloses the subject matter recited in 
claim 16 when the threshold amount is chosen to be one. Applicant disagrees. 
MR does not teach or describe choosing between two types of matrixes or doing 
so based on feature vector length or any number of previously retrieved images. 

Further, using MR's notation, claim 16 recites that n > N + threshold 
corresponds to the first type of matrix. The second matrix type would be used 
when n < N + threshold. 

MR, in contrast, teaches that the singular covariance matrix case arises 
when the number of feedback points N is less than the number of feature 
dimensions n, i.e., N < n and teaches use of either one of two other types of 
matrixes (Calcs. 6 and 7, or Calc. 8) otherwise, that is, when N >n. Such does not 
correspond to use of any threshold in addition to a number of datapoints in making 
any decision. 

Claim 17 depends from claim 16 and recites that "the first type of matrix 
comprises a full matrix and the second type of matrix comprises a diagonal 
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matrix 11 , which is not taught or disclosed by MR. As noted above, MR fails to 
teach or disclose the subject matter recited in claim 16. As also noted above, MR 
fails to teach or disclose choosing between diagonal or full matrices based on a 
number of prior samples. 

Claim 18 recites "One or more computer readable media including a 
computer program that is executable by a processor to perform the method recited 
in claim 11", which is not taught or disclosed by MR. MR is completely silent as 
to how the ideas described by MR might be implemented and does not provide any 
description regarding implementation whatsoever. Examples of approaches that 
might be employed could include an ASIC or neural network or an analog 
computer, none of which represent computer readable media or a computer 
program that is executable by a processor. 

In other words, MR does not teach or disclose any computer readable ; 
media including a computer program that is executable by a processor. Put 
another way, MR is completely void of the language "computer readable media" 
or "computer program". 

In contrast to MR, claim 46 recites "A method of generating a query vector 
to compare to a feature vector of another image , the method comprising: receiving 
feedback regarding the relevance of each image of a set of images ; wherein TV 
represents the number of images in the set of images for which user relevance 
feedback has been received, n n represents the relevance of image n in the set of 
images, k represents a transposition of a vector generated by concatenating the 
individual n n values, and X represents an image matrix that is generated by 
stacking N training vectors corresponding to the set of images into a matrix; and 
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generating a query vector ( q ) corresponding to one of a plurality of features as 
follows: 

? =- > 

which is not taught or disclosed by MR. 

MR teaches (with respect to theorem 1) that a query vector = X T (goodness 
values)/(sum of goodness values), where X represents the entire data point matrix 
(see Table I). This is not the same as the recitation of claim 46, which provides 
that the query vector is given as the transpose of a vector formed by concatenating 
relevance values times an image matrix formed by stacking N training vectors all 
divided by a sum of the relevance values. 

Further, there is no teaching or disclosure in MR of searching images, or of 
"receiving feedback regarding the relevance of each image of a set of images", as 
recited in claim 46. 

Dependent claims 9, 12-18, 35-38, 40, 42-44, 47, 49 and 51 are allowable 
as depending from an allowable base claim and for their own recited features 
which are neither shown nor suggested by the prior art. 

RFT fails to provide the elements recited in Applicants' claims 1, 8, 9, 32, 
35-38, 40-44 and 48-51, and MR fails to provide the elements recited in claims 11- 
18, 46 and 47, as required by MPEP §2121, see item (i) above in "Anticipation is a 
legal term of art". As a result, neither RFT nor MR can set forth these respective 
elements as they are recited in the respective claims, as required by MPEP §2131 
(item (ii), supra). Accordingly, the teachings of the references must be modified 
in order to arrive at the subject matter of the claims, in contravention of MPEP 
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§706.02 (item (iii), supra). Additionally, because neither RFT nor MR meets the 
criteria for a finding of anticipation outlined above, neither RFT nor MR cannot 
enable the subject matter recited in the respective claims, and thus the references 
and the arguments in the Office Action fail to meet the criteria of MPEP §2121.01 
(item (iv), supra). Inasmuch as none of the requirements for a finding of 
anticipation are met by the arguments presented and the references, the 
anticipation rejections of claims 1, 8, 9, 11-18, 32, 35-38, 40-44 and 46-51 are 
defective and should be withdrawn, and claims 1, 8, 9, 11-18, 32, 35-38, 40-44 
and 46-51 should be allowed. 
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35 U.S.C. § 103 

In contrast to RFT, claim 10 recites "One or more computer readable media 
as recited in claim 1, wherein the low-level features include: a color moments 
feature, a wavelet based texture feature, and a water-fill edge feature", which is not 
taught, disclosed, suggested or motivated by RFT. As noted above with respect to 
claim 1, RFT is silent with respect to computer readable media having stored 
thereon a plurality of instructions that, when executed by one or more processors, 
causes the one or more processors to ...." RFT is silent with respect to 
implementation. RFT is also silent with respect to any "water-fill edge feature", as 
recited in claim 1,0. The discussion of Calcs. 1 and 2, cited in the Office Action 
(p. 10) is devoid of any mention of "a wavelet based texture feature". 

The Office Action states (p. 10) that the Examiner takes Official Notice that : 
such representations were of common practice. Additionally, the Office Action ,. 
admits, on the record, that the reference fails to supply all aspects of the subject 
matter as recited in the claim. 

Specifically, the Office Action admits (p. 10) that RFT fails to disclose 
"that the texture feature includes "a wavelet-based texture feature" and that the 
shape feature includes "a water-fill edge feature" as claimed" and then states that 
"However, this is only because RFT is silent on specific examples for 
representations of texture and shape within this particular article." Applicant notes 
that RFT provides numerous specific examples (see, e.g., Calc. 23 and text 
associated therewith, listing color histogram, color moments, Tamura, co- 
occurrence matrix, Fourier descriptor, chamfer shape descriptor; see also list of 
texture feature representations on the second page of the article). 
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The Office Action further states (p. 10, bottom, continuing on p. 11) that 
"The examiner takes Official notice that wavelet based texture features were 
texture representations of common practice in the art at the time the invention was 
made, and further that water-fill edge features were shape representations of 
common practice in the art at the time the invention was made." 

Claim 52 recites "One or more computer readable media as recited in claim 
1, wherein the receiving feedback comprises receiving feedback via speech 
recognition", which is not taught, disclosed, suggested or motivated by RFT. 

The Office Action states (p. 11) that "RFT does not explicitly disclose that 
the receiving feedback comprises speech recognition as claimed. This however, is 
only because RFT is silent on the computerized means for specifying the user 
feedback. The examiner takes Official notice that the use of speech recognition, 
for user input/feedback was common practice ...." 

With respect to both claims 10 and 52, inasmuch as these allegations are 
not supported by the reference, they must be based on either (i) personal 
knowledge of the Examiner or (ii) hindsight reconstruction based on Applicants' 
own disclosure, employing an "obvious to try" standard of unpatentability. Either 
is improper as is discussed in further detail below. 

Applicant notes the requirements of MPEP §707, entitled "Examiner's 
Letter or Action". This MPEP section states, in subsection (d)(2), that: "When a 
rejection in an application is based on facts within the personal knowledge of an 
employee of the Office, the data shall be as specific as possible, and the reference 
must be supported, when called for by the applicant, by the affidavit of such 
employee , and such affidavit shall be subject to contradiction or explanation by 
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the affidavits of the applicant and other persons." Accordingly, Applicant calls for 
such affidavit in the event that the Examiner persists with the rejection of claims 
10 and 52 based on the personal knowledge of the Examiner. 

As there is no basis for the Examiner's contentions within the cited 
references, the only possible motivation for these contentions is hindsight 
reconstruction wherein the Examiner is utilizing Applicant's own disclosure to 
construct a reason for modifying and adding to the disclosure of the cited 
reference. The Examiner is reminded that hindsight reconstruction is not an 
appropriate basis for a §103 rejection. (See, e.g., Interconnect Planning Corp, v. 
Feil, 227 USPQ 543, 551 (Fed. Cir. 1985); In re Mills, 16 USPQ2d 1430 (Fed. 
Cir. 1990) (explaining that hindsight reconstruction is an improper basis for 
rejection of a claim).) 

Further, suggestion to modify as put forth in the Office Action appears to 
employ an improper "obvious to try" rationale, as is discussed below in more 
detail with reference to MPEP §2145(X)(B). This MPEP section states that: 

The admonition that 'obvious to try* is not the standard under §103 
has been directed mainly at two kinds of error. In some cases, what 
would have been 'obvious to try 1 would have been to vary all 
parameters or try each of numerous possible choices until one 
possibly arrived at a successful result, where the prior art gave either 
no indication of which parameters were critical or no direction as to 
which of many possible choices is likely to be successful .... In 
others, what was 'obvious to try' was to explore a new technology or 
general approach that seemed to be a promising field of 
experimentation, where the prior art gave only general guidance as 
to the particular form of the claimed invention or how to achieve it. 
In re O'Farrell, 853 F.2d 894, 903, 7 USPQ2d 1673, 1681 (Fed. Cir. 
1988) (citations omitted). 
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No guidance has been identified within the reference to determine which 
elements to pick or choose from the reference, or of how to couple them to 
somehow arrive at subject matter such as is claimed (see also discussion of MPEP 
§2143 infra). 

Additionally, the admissions, on the record, that the reference fails to teach, 
disclose, suggest or motivate the subject matter recited in claims 10 and 52, show 
that a prima facie case of unpatentability has not been established (see discussion 
of MPEP §2143 below). 

In contrast to RFT, claim 2 recites "One or more computer readable media 
having stored thereon a plurality of instructions that, when executed by one or 
more processors, causes the one or more processors to perform acts including: 
receiving an initial image selection; generating a plurality of query vectors by 
extracting, for each , query vector, one of a plurality of low-level features from the 
initial image selection; selecting a set of potentially relevant images based at least 
in part on distances between the plurality of query vectors and a plurality of 
feature vectors corresponding to low-level features of a plurality of images; 
receiving feedback regarding the relevance of one or more images of the set of 
potentially relevant images; generating a new plurality of query vectors based at 
least in part on the feedback; generating a weighting of feature elements based at 
least in part on the feedback ; and selecting a new set of potentially relevant images 
based at least in part on both the weighting of feature elements and distances 
between the new plurality of query vectors and the plurality of feature vectors , 
wherein the selecting a new set of potentially relevant images comprises using a 
matrix in determining the distance between one of the new plurality of query 
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vectors and one of the plurality of feature vectors , and further comprising 
dynamically selecting the matrix based on both a number of images in the set of 
potentially relevant images for which relevance feedback was input and a number 
of feature elements in the one feature vector ", which is not taught or disclosed by 
RFT. The rejection of claims 19 and 20 is stated (p. 14) to be on the same basis as 
the rejection of claim 2. 

Neither reference provides any teaching of " generating a new plurality of 
query vectors based at least in part on the feedback; generating a weighting of 
feature elements based at least in part on the feedback ; and selecting a new set of 
potentially relevant images based at least in part on both the weighting of feature 
elements and distances between the new plurality of query vectors and the 
plurality of feature vectors ", as recited in claim 2. RFT teaches use of a fixed (and 
apparently predetermined) set of feature vectors f h together with a set of weights 
that are the sole representation of feedback. MR teaches selection of a new query 
point q and does not teach generation of weighting elements based on feedback. 

As a result, neither reference provides any teaching of " using a matrix in 
determining the distance between one of the new plurality of query vectors and 
one of the plurality of feature vectors ", as recited in claim 2. 

As noted in the Office Action (p. 12), RFT is silent with respect to selection 
of a matrix. RFT also fails to provide any criteria for such selection. The Office 
Action indicates that motivation to take the teachings of MR regarding matrix 
selection would be "to avoid the problem of a singular and non-invertible 
covariance matrix, as disclosed by MR." 
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However, RFT provides experimental results (§4, Figs. 3 and 4). The ideas 
disclosed by RFT do not encounter any such problem or issue. Such is non 
sequitur to anything contained in or disclosed by RFT. If such were a problem 
with the methods taught by RFT, RFT would have adopted techniques to address 
such. The fact that RFT didn't develop and adopt such techniques shows that this 
problem and such motivation are absent for the techniques disclosed by RFT. 

The Office Action further states (p. 1 1) that further motivation lies in MR's 
building off of the MARS system and that such is explicitly referenced in RFT. 
Such does not comprise motivation. 

MR mentions the MARS system as "Related Work" (section 2), stating that 
such is a query point movement system (section 2.1), which "directly applied the 
Rocchio's formula" and that such also proposes re- weighting (section 2.2). MR 
explicitly states (bottom, left side of p. 220) that their approach "does not use ad- 
hoc heuristics (such as jS and y in the Rocchio's formula". MR also explicitly 
states (Abstract; section 2.2; section 6)that the approach set forth by MR offers 
significant advantages that no other system can provide when compared to other 
systems. 

MR explicitly states (p. 220) that the technique proposed by MR includes 
the query refinement technique of MARS as a special case but presents significant 
advantages (labeled (a) through (c)) relative to prior techniques. As such, MR is 
aware of the MARS project and certainly does not provide motivation to combine 
the teachings of RFT with those of MR. 

Both references use a number of terms and labels in common - e.g., 
"vector", "database" etc. In fact, both references are written in the same language. 
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None of these facts are remarkable or constitute a basis for motivation to combine 
teachings or to modify teachings. 

Additionally, claim 2 recites "using a matrix in determining the distance 
between one of the new plurality of query vectors and one of the plurality of 
feature vectors". In other words, a hierarchical approach is taken to determining 
how closely two images match: first individual distances between the feature 
vectors and the query vectors are determined and evaluated, and then these 
individual distances are combined. 

RFT does not describe such an hierarchical process. RFT describes an 
approach whereby an object's similarity to a query object is evaluated (step 4), the 
representation's similarity values are combined into a feature's similarity value 
(step 5) and these similarities are then combined (step 6) to provide an overall 
similarity. This overall similarity is then employed to evaluate rank ordering (step 
7). 

In contrast to Applicant's claims, MR describes a process that also is not 
hierarchical - as shown by Calc. 4, which seeks to minimize sums over all N of the 
samples, of (Calc. 3) sums over all of the query vector elements and feature vector 
elements. There is no guidance contained in the references to show how to adapt 
the teachings of RFT to arrive at the hierarchical process described by claim 2, or 
of how to fold in selected bits and pieces of MR into the framework of RFT to 
somehow attempt to arrive at the subject matter recited in claim 2. 

MR uses a matrix M to describe a generalized distance (see section 3.2). 
RFT does not describe any matrix with respect to steps (1) through (10). As a 
result, there is no motivation anywhere in RFT to import a matrix from any 
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reference to attempt to arrive at Applicants' claimed subject matter. As such, the 
proposed combination fails to provide the subject matter recited in any of claims 2 
or 19. 

Claim 3 recites "One or more computer readable media as recited in claim 
2, wherein the dynamically selecting comprises using a diagonal matrix if the 
number of images in the set of potentially relevant images for which relevance 
feedback was input is less than the number of feature elements in the one feature 
vector , and otherwise using a full matrix ", which is not taught, disclosed, 
suggested or motivated by the cited references, alone or in any proper 
combination. 

The Office Action states (pp. 12, 13) that MR, as added to RFT, teaches the 
dynamically selecting with respect to claim 3 (p. 12) and that (p. 14) claims 2 1 and 
22 are rejected on the same basis as claim 3. The discussion of application of MR 
and RFT to attempt to arrive at the subject matter of claim 3 relies on the rejection 
of claims 14 and 15. As noted above with respect to claims 14 and 15, RFT fails 
to provide any basis for discrimination between use of a full matrix and use of a 
diagonal matrix. Further, the Office Action is void of any discussion of how or 
why the teachings of MR might be combined with those of RFT. Clarification is 
requested. As such, the proposed combination fails to provide the subject matter 
recited in any of claim 3 or claims 21 and 22. 

Claim 4 recites "One or more computer readable media as recited in claim 
2, wherein the dynamically selecting comprises: if the number of images in the set 
of potentially relevant images for which relevance feedback was input is not less 
than the number of feature elements in the one feature vector , then using one 



lee®hayes c* k»-324.«2m 



54 



Application No. 09/660,536 




matrix that transforms the query vector and the one feature vector to a higher-level 
feature space and then using anbther matrix that assigns a weight to each element 
of the transformed query vector and the transformed feature vector ; and if the 
number of images in the set of potentially relevant images is less than the number 
of feature elements in the one feature vector, then using a matrix that assigns a 
weight to each element of the query vector and the one feature vector ", while 
claim 23 recites "A method comprising: generating a query vector corresponding 
to a feature of one image ; identifying a feature vector corresponding to the feature 
of another image; identifying a number of training samples for which relevance 
feedback has been received; if the number of training samples either equals or 
exceeds a threshold amount, then determining a distance between the query vector 
and the feature vector including transforming the query vector and the feature 
vector to a higher-level feature space and then assigning a weight to each element 
of the transformed query vector and the transformed feature vector ; and if the 
number of training samples does not exceed the threshold amount, then 
determining the distance between the query vector and the feature vector including 
assigning a weight to each element of the query vector and the feature vector " and 
claim 30 recites "A system comprising: a query vector generator to generate a 
query vector corresponding to a feature of one image; a comparator, coupled to the 
query vector generator, to, identify a feature vector corresponding to the feature of 
another image, identify a number of training samples for which relevance 
feedback has been received, if the number of training samples either equals or 
exceeds a threshold amount, then to determine a distance between the query vector 
and the feature vector including transforming the query vector and the feature 
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vector to a higher-level feature space and then assigning a weight to each element 
of the transformed query vector and the transformed feature vector , and if the 
number of training samples does not exceed the threshold amount, then to 
determine the distance between the query vector and the feature vector including 
assigning a weight to each element of the query vector and the feature vector ", 
which is not taught, disclosed, suggested or motivated by RFT or MR, alone or in 
any proper combination. 

The Office Action states (pp. 12, 13) that MR, as added to RFT, teaches the 
dynamically selecting of claim 4 (p. 13) and that (p. 14) claims 23, 24, 29 and 30 
stand rejected on the same basis as claim 4. 

The discussion relative to claim 4 of application of MR and RFT to attempt 
to arrive at the claimed subject matter focuses on section 3.3 of MR. However, 
that portion of MR discusses minimization of a distance function as shown in 
Calc. 4. In each case, whatever transformation of query vectors takes place 
involves the same dimensions of feature space and the same weighting procedures. 

Additionally, MR describes use of one matrix M under one set of condition, 
and another matrix M otherwise. There is no description in MR, RFT or the 
Office Action of " transforming the query vector and the feature vector to a higher- 
level feature space", followed by " then assigning a weight to each element of the 
transformed query vector and the transformed feature vector", as recited in claims 
4, 23 and 30. 

Further, the Office Action is void of any discussion of how or why the 
teachings of MR might be combined with those of RFT. Clarification is 
requested. 
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MR, with or without RFT, changes the choice of matrix M, but does not 
provide any description of transformation and weighting of query and feature 
vectors and thus cannot possibly provide criteria for when such transformation 
should be applied, as recited in claims 4, 23 and 30. As such, the proposed 
combination fails to provide the subject matter recited in any of claim 4 or claims 
23, 24, 29 and 30. 

Additionally, there is no guidance in the references as to how to adapt the 
teachings of RFT to arrive at the claimed subject matter or of how to fold in 
selected bits and pieces of MR into the framework of RFT to somehow attempt to 
arrive at the subject matter recited in claim 4, 23 or 30. 

Claim 5 recites "One or more computer readable media as recited in claim 
2, wherein X represents an image matrix that is generated by stacking N feature 
vectors, each of length K, corresponding to the set of potentially relevant images : 
for which relevance feedback was received and resulting in an ( N x K ) matrix, C 
represents a weighted covariance matrix of X , det(C) represents the matrix 
determinant of C, and the matrix comprises a full matrix (W*) that is generated 
as follows: W* = (det(C))*C _I , while claim 7 recites "One or more computer 
readable media as recited in claim 1 , wherein TV represents the number of images 
in the set of potentially relevant images for which relevance feedback has been 
received, n n represents the relevance of image n in the set of images, n 
represents a transposition of a vector generated by concatenating the individual n n 

values, and X represents an image matrix that is generated by stacking AT training 
vectors corresponding to the set of potentially relevant images into a matrix , and 
wherein each new query vector (q) of the new plurality of query vectors is 



lee@hayes pic 509-324^256 



57 



Application No. 09/660,536 



generated as follows: q = " and claim 55 recites "One or more computer 

readable media including a computer program that is executable by a processor to 

cause the processor to perform acts of: receiving user feedback regarding the 

relevance of each image of a set of images, the user feedback forming a range 

including at least Highly Relevant, Relevant, No Opinion, Irrelevant, and Highly 

Irrelevant; wherein TV represents the number of images in the set of images for 
which user feedback has been received, K n represents the relevance of image n in 

the set of images, n represents a transposition of a vector generated by 
concatenating the individual n n values, and X represents an image matrix that is 

generated by stacking TV training vectors corresponding to the set of images into a 

— > 

matrix; and generating a query vector ( q ) corresponding to one of a plurality of 

-> TC X ~ 

features as follows: q = which is not taught, disclosed, suggested or 

71=1 

motivated by the cited references. Claim 55 is stated (p. 15) to be rejected on the 
same basis as claim 7. 

As noted above with reference to claim 46, MR does not teach or disclose 
stacking N feature vectors corresponding to the set of potentially relevant images 
for which relevance feedback was received, or any (NxK) matrix. MR teaches 
(with respect to theorem 1) that a query vector = X (goodness values)/(sum of 
goodness values), where X represents the entire data point matrix (see Table I). 
This differs from the recitation of claims 5, 7 or 55, which provide that 
concatenating relevance values times an image matrix formed by stacking N 
training vectors provides an image matrix X , or that " C represents a weighted 
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covariance matrix of X , det( C ) represents the matrix determinant of C , and the 
matrix comprises a full matrix (W*) that is generated as follows: 

W* = (det(C))*C _1 as recited in claim 5, or that "each new query vector (q ) of 

-> ji X 

the new plurality of query vectors is generated as follows: q = — — as recited 
in claims 7 and 55. ^ 

The Office Action offers (p. 13) Theorem 2 of RFT as providing the subject 
matter of claim 5. Claim 5 refers to a full matrix. Theorem 2 is explicitly devoted 
to a diagonal matrix of variances Oj . 

The Office Action is void of any discussion of how or why the teachings of 
MR might be combined with those of RFT. Clarification is requested. As such, 
the proposed combination fails to provide the subject matter recited in any of 
claims 5, 7 or 55. 

Further, there is no teaching or disclosure in MR of searching images, or of 
"receiving feedback regarding the relevance of each image of a set of images", as 
recited in claims 7 and 55. MR is silent with respect to searching images. 

Applicant notes the requirements of MPEP §2143, entitled "Basic 
Requirements of a Prima Facie Case of Obviousness" (see also MPEP §706.02(j), 
entitled "Contents of a 35 U.S.C. 103 Rejection.") . MPEP §2143 states that "To 
establish a prima facie case of obviousness, three basic criteria must be met. First, 
there must be some suggestion or motivation, either in the references themselves 
or in the knowledge generally available to one of ordinary skill in the art, to 
modify the reference or to combine reference teachings." 

Inasmuch as the references fail to teach or disclose the elements recited in 
the claims, the references cannot provide motivation to modify their teachings to 
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arrive at the invention as claimed, and the Examiner has identified no such 
teaching or disclosure in the references. As a result, the first prong of the test 
cannot be met. 

MPEP §2143 further states that "Second, there must be a reasonable 
expectation of success. Finally, the prior art reference (or references when 
combined) must teach or suggest all the claim limitations." 

Inasmuch as the references fail to provide all of the features recited in 
Applicant's claims, the third prong of the test is not met. For example, as noted 
above, the references are silent with respect to computer readable media, as recited 
in claims 2-7, 19-22, 29 and 53-56. As a result, there cannot be a reasonable 
expectation of success. As such, the second prong of the test cannot be met. 

MPEP §2143 additionally states that "The teaching or suggestion to make 
the claimed combination and the reasonable expectation of success must both be;, 
found in the prior art, not in applicant's disclosure. In re Vaeck, 947 F.2d 488, 20 
USPQ2d 1438 (Fed. Cir. 1991)." This criterion cannot be met because the 
references fail to teach or disclose the elements recited in the claims. 
Accordingly, the unpatentability rejections based on RFT, with or without the 
teachings of MR, fail all of the criteria for establishing a prima facie case of 
obviousness as set forth in the MPEP. 

Moreover, no evidence has been provided as to why it would be obvious to 
modify and/or combine the teachings of these references. Evidence of a 
suggestion to combine may flow from the prior art references themselves, from the 
knowledge of one skilled in the art, or from the nature of the problem to be solved. 
However, this range of sources does not diminish the requirement for actual 
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evidence . Further, the showing must be clear and particular . See In re 



Dembiczak, 175 F.3d 994, 998 (Fed. Cir. 1999). 

For at least these reasons, Applicant respectfully requests that the §103 
rejections be withdrawn, and that Applicant's claims 1-30, 32-38, 40-44 and 46-56 
be allowed. 

Conclusion 

Claims 1-30, 32-38, 40-44 and 46-66 are in condition for allowance. 
Applicant respectfully requests reconsideration and issuance of the subject 
application. Should any matter in this case remain unresolved, the undersigned 
attorney respectfully requests a telephone conference with the Examiner to resolve 
any such outstanding matter. 



Respectfully Submitted, 





Frederick M. Fliegel 
Reg. No. 36,138 
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